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.ABSTRACT 

The purpose of this study was to evaluate the on-site 
evaluation phase of the three-phase system for statewide evaluation 
of occupational education programs in Illinois. The specific problem 
was to explore the nature of selected changes in the occupational 
education programs of comprehensive secondary schools over five years 
and the extent to which these changes may be related to the on-site 
evaluation process. Data were collected on the fifteen comprehensive 
secondary school districts in the sample through two guestionnaires 
Administered to selected local education agency (LEA) personnel and 
the on-site evaluation. Among the findings were that (1) both LEA 
personnel and on-site evaluation team members agreed that 'there had 
been improvement in all occupational education program components and 
that the on-site evaluation had contributed to that improvement; (2) 
both LEA personnel and team members reported that the on-site 
evaluations were a beneficial and worthwhile investment of resources, 
and (3) LEA personnel and on-sii:e team members agreed as to the 
relative improvement in <aach of the program components. (LRA) 
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Preface 



This study was undertaken as a part of the on-going program of research, 
teaching and service of the Department of Vocational and Technical Education 
at the University of Illinois, It represents a. pai't of the continuing 
cooperative efforts among the University, the Department of Adult, Vocational 
and Technical Education (DAVTE), Illinois Office oi; Education, the local 
educational agencies and the occupational educators of Illinois, o 

The University had had a contract to assist in the carrying out of the 
Three Phase System (TPS) of Evaluation since its inception. Much of the 
data of record are housed at the University. Ms 'AnCione tte Wirth was 
associated with this TPS activity since the early years of its operation 
and pro' ided essential assistance in several aspects of the study. 

The entire study was undertaken with combined purposes: to provide one 
aspect of evaluation of the TPS. and to serve as the mechanisms for a doctoral 
dissertation for Mr. Terry R. Smith. This combined approach provides both 
immediate benefits to th'^ field through the research reports and related 
presentations and long- cm benefits through the experiences gained by 
professional personnel in the ^ield. 

The study resulted in a more detailed and comprehensive Technical ^ 
Report (which was also the doctoral dissertation) this Executive Summary, 
and supplemental information which can be utilized by DAVTE in reviewing 
/the TPS. A limited number of copies of the complete Technical Report have 
/been provided to DAVTE; copies of the dissertation are on deposit at the 

University of Illinois library or'available through University Microfilme and 
\ the ERIC system. The Technical Report includes detailed descriptions of the 
analyses, procedures and research instruments as well as a number of tables 
of data which were not fully analyzed for this scudy. 

» 

The project staff takes this opportunity to express appreciation to 
ttiose local school districts and their personnel, the members of the on-site 
evaluation teams, the DAVTE Regional Directors and others who gave of 
their time and efforts so that this study could be carried out. A •Special 
recognition is given to the members of the study a^dvisory committeef Peter 
Johnson, Warrt^n Collier, Harold Finn and John Washburn. 

It is hoped that this study will make a contribution to the occupational 
education students who are the ultimate reason for the existence of the ; 
occupational education programs and their personnel at all levels. 



Robert M. Tomlinson 
Project Director 
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^ ON-SITE 'EVALUATION OF OCCUPATIONAL . ' 

Program evaluation at the local, state anfl federal levels is being 

i' 

given increased importance. Accrediting agencies, local, state and 

national organizations and governmental agencies at every level are demandi 

) . 

that ,the educational agencies and institutions provide reports of program" ' 
evaluation whicl^emonsttate accountability for the funds received and" ' 

the clientele served. , ^ v . 

' ■ ' ' .... 

In response to the need for greater accountability, state and federal 
education agencies have implemented evaluation requirements into theix? 
funding pblicies and' procedures • These requirements have beefi developed 
to allow the state and 'federal education agencies to obtain data on the 
quantity and quality of the educational programs offered 'at the local and 
state- levels. It wa6 the intent'that witji 'these data it would be"^possible 
for the state and federal education agencies to make decisions concerning 
the present programs* an*, to plaft future programs, funding, and reseaxdh 
activities. * • ' • ' 

Evaluation in Occupational Education ' 

The area of occupational ^education has been extensively involved in - 
the mo^c towards greater program evaluation and providing a'ccountability 
in^^'ormation to state and federal education agenciciS. Federal legislation 
enacted in 1963 (Vocational Education Acc oi: 1963 , PL-88-210) required the 
evaluation of ail funded vocational education programs and activities; this 
Act marl/ed the beginning ot* the formal process oi evaluating vocational ' 
education programs • * • ■ 



The structure for evaluating vocational education was further expanded 
in 1968. The' Vocational Education Amendri^ent§ of 1968 (Public Law 90-576) 
required each state to participate in the planning and evaluation of the 

vocational education programs within their state. 

. ■ \ 

Included as 'Title II of the Education Amendments of 1976 (?ubUc Law 
94-482) are the Vocational Education Amendments of '1976 which further^ 

extend the r^pquirements fjor evaluation of vocational education. To be 

I • 

eligible to receive. federal vocational fund?, the Vocational Education 
Amendments of 1^76 required that state Departments of Vocational Education 
agencies provide the US Office of -Education with more specific data and on 
additional components of the local and state vocational education programs 
tshan previously were required by the Vocational Education Amendments of 
1968. 

On-Site Evalixation ' , 

• • r 

Several states have implemented statewide evaluation systems, in meeting ' 
these evaluation requirements, which include an on-site, evaluation component 

as a part of the evaluation process. These on-site evaJ.uations are made by 

~^ ' 

evaluation teams which may differ in structure and composition from sJ:ate to 
state. In some states the team is composed of state vocational educatioxv. 
department personnel. In other states t"he team is composed of Iqrcal 
educatioi^ agency (LEA) isersonnel ins true tors, guidance personnel and 
administrators), business, industry and labor personnel, and present or 
former occupational education students. Some states use a combination 
of these pattems, including both stat<}2 personnel and .local personnel on 
the on-sike evaluation team. \ 



The Occupational Education Evaluation System in lit'..* 

* . t ♦' I 

t • - • 

s I . * . 

\X^?1^^69, the Illinois State Board of Vocational Education and 
'Rehabilitation developed provisions for a statewide 'evaluat^n system 
as . a part of their State Plan^." "The statewide evaluation system was 
^esigneq to ^ovide for f.he evaluation of all local vocational education 
. prog^:ams, ser\(i(jes and activities in terms of meeting the needs, interests 
and abilities of hyi^vidual students. , The system was cal..led the Three 
I'hase System for Statewide Evaluation of Occupational Edvxation programs 

« 

(TPS). ' V 

' AsJ;the title implies, the evaluation system ciev« loped in Illinois is 
comprised of three phases: (1) local district planning of the occupational 
education program which culminates in a written One and Five Year Plan for 

Occupational Education for the LEA (this plan must be up-dated each year),' 

■ ^ ' • ■ 

(2) an annual review of the LEA's plan by. the staff of the 111 Alois , 
.Department cf Adult, Vocational and Technical Education (DAVTE), and (3). an 
on-site evaluation once .each five- years ^h^ch is conducted "^by a team of 
individuals not associated with the LEA being evaluated or DAVTE (Wentling 
and Klit, 1972). - • , ' 



i 



'^^ Purpose of the Study 



Illinois has expended large amounts, o;^ time and money in developing and 
carryj^ng out its TPS which. Includes an on-site evaluation phase. This 
system was developed to evaluate the existing, complete occupational 
education program and to provide recommendations to the local and state 
oducation^agencies for improvement of the local occupational education 
program. 



iThe results of thin study provide a basis for recommendations. 'Which 
will be directed toward improvement of the on-site evaluation phase and 

to obtain, a greater return on 'the investment being made in the on-sit^ 

i> . ■ ' ■ ■ ■ 

tM/aluation phase of tt%e IlljL(aois TPS. findings j*ill also be of benefit 

to other states using a similar on-site evaluation procedure. 

Statement of the Problem * 



The problem of this study was to explore the nature of selected ^ 
changes in the occupational education programs of comprehensive, secondary 
schools over a period of five years, and the extent to which these changes 
may lie related to th^ on-site evaluation process. 

The nature of selected ch^in^fes in »the occupational education programs 
was explored. by the use of information which was obtained at three 
different periods >5f. cime. Data collected during a first on-site- ev4luati,on 
provided a baseline. Data obtained at a similar, "second on-site evaluation 

♦ 

conducted five years latter provided a second source of data from the same. 

LEAS.-. The third source of data was from follow-up questionnaires 
, admin is.te red to sel^^ed LEA personnel and 'to each member of the on-site 

evaluation teams that was involved in the second on-site^ visit, to the 

selected LEAs. , 

Some of the information selected to explore the nature of changes in ^ 

the occupational, education programs came from the following, existing data 
./3f record whi^h were obtained as a part of the first and second on-site 
^visits: ' the Preliminary Evaluation Instrument - Personnel (PEI-P) and 

Preliminary^ Evaluation Instrument - Student (Pfll-S) data, the Evaluation 

« 

^ Team Reports (ETRs), and the One and Five Year Plans submitted by the i,EAs 

I 

to tht' std.te QAVTE. ... 



To explore possible reUtionships boCwccn tiie Se'lecPed changes under 
• investigation and the on-site evaluation phase, information ailso was 
obtained 'questionnaire, from selecced LEA personnel and from theNnembers 
04 the on-site evaluation taam who participated in the second on-site 



visits The sclactad LEA personnel included oc'cu^ational instructors, 
guidance personnel and occupational education a^minigtrafors • Each was . 
asked to provtd^inf ormati9n relative to his/hejr perceptions of the extent 
to which changes in the occupational fduoation programs were related to 
the: on-site evalyatipn ^process, Each also' was *asked j:6 cit'p *benef its 
gained from the on-site evaluation. -In addition, the members, of the 
respettive second ,on-5ite evaluation teams were/asked to provide 
informatian relative to^ their perceptions of the eVtent to which they found 
changes in the occupational education programs related to the on-site " ' 

evaluation process and to cite benefits that t4iey, individually, may have 

^ ^ ' • • ■ ' . , " , *^ ' . 

* . *^ * • » 
received by. serving as a team member ^ • . * • . 

* ' ' ' ^The Population' and §araple. - y 

• The population for this study consisted of all ^public comprehensive 
secondary ischools ^n the state^j of Illinois offering an occupational 
education program. From this population a sub-population was identified 
■iwhich consisted of approximately iifty comprehensive secondary schools, ^ 
This sub-population was composed of all such schools which had received 
their first: on-site visit in 1972-1973. and their second on-site evaluation 
during the 1977-1978 scliool year, • • ^ 

The sample for this study was comprised of fifteen comprohfensive 
socondary school districts. This samplt* vas soLt»ctt5d from tho fifty 



ulibible,' comprehensive secondary school, dis Criccs<which had received ' 
ChQi'Asecond on-site; evaluation during the 1977-1978 school year. Xhe 
sample was stratified to' include various ^sizes of' LEAs from, the various 

regions of the state and to provide a representative sami)le' of these LEAs» 

" . ' " ' ' ^ /" ■ ■ 

Two LEAs were selected from each o.f the sevea Illinois Office of 

Education -regions, outside the city of Chicagp and <Jne remaining LEA .was 

selected at large. 'This selection process facilitated * the selection of 

LEA« *which represented the various gisographic areas within the^state.. 

Within ^.ch region, the Seligible LEAs were grouped total student 

, population at thee secondary level into t:wo classif ications^ tho^se with a 

larger than median enrollment anri those with a smaller than median 

^ enrollment wi|hin that region; . One LEA from each si^e classification was 

selected for the study. -One additional LEA .was selected from tho^e LEAs 

that were not previously selected. This procedure facilitated the 

''selection of LEAs which represented the various sizes of* scjiools within 

\he state. . ^ \ ' . 

' Data Collection Procedures ^ - * 

■' To. accomplish this investigation, the following procedures were. 
* Utilized: . ' \ ^ , • . " " ■ 

1. Two questionnaires were developed to obtain. data from 
selected LEA .personnel and on-site evaluation team 
members after the second on-site ^valuati^on. 

2. Selected data of recprd were obtained from DAVXE records . 
» pf student and LEA personnel responses at. the. time of 

f , the first and second on-site evaluations. These data ^ 
.01 record' included: PEI (student and personnel) data / 



; 'I ■ » 

and the ETRs fo,r the first and second " on-site evaluations. ' 

« > 

3. The data obtained from the selected LEA persojanel, , ^on-site 

evaluation tea* members and the data of record were aualyaed 

J ' ' * 

J ^ . in response to the research questions formulated for the 

J;{ study. • , ' — * \ ... 

** * , * » ' * ' ' 

^ Data Collection by the Fo11ow-Up Questionnaires 

f 

Da'ta collectioii.by the two fallow-up questionnaires began on April 
20, 1978 and contihued until June 1, 1978. The time schedule for data 
collection involyed the initial mailing and two follow-up mailings at', two 
week inter\^als. Two weeks after the initial malllrlg, those individuals 
who had not returned their , follow-up queswionnaire received a foUow-up 
letter, a second copy qif the follow-up questionnaire and another return 
addressed, stamped envelope. •'• ."'^ '. ' ^ ' 

Two weeks luter, the third mailing was m4de to those, individuals who" 

still had not. responded to the initial or .second mailing. This final, 

... . ' t . ■ • _ 

follow-up also contained a follow-up letter, ano.her copy of the foll-dw^up 
questionnaire and another return addressed, stamped envelope. Six weeks, 
after the initial mailing, the data collection by follow-up questionnaire 
was completed. . ^ 

Collection of Data of Record \ 

•^he data of record utilised for the study had 'been collected at two 

.... ' ' ''i 

points in time, five years apart. These data were \:ollected as a part of 

dhe on-site evaluation phase of the TPS*.. Data for the selected LEAs from 

both the first ..id second on-site evaluations were used in this *study. The 

data of record inclLde tht* PEI data for the stu:!ents and faculty, the 



evaluation team report and. a Team On-Site, Evaluation Instrument. The 



Team Ofiv?5ite Evaluation Inatrumeilt^xas "completed by the team leader 
with input provided fr.om the te«»&i members. This was usually completed 
after the evaluation team report was completed and prikir to an exit 
interview. ^ 



Descriptive Overview .of the Sample 



■ The sample" for this study included the following groups associated 
with the fifteen LEAs: occupational edu9ation students, LEA 'occupational 
education" persbnnel and on-site evaluation team members. Data were \ 
^obtained, frc-.i occupational education student the time of the first 
and second on-^ite evaluations. Data also w.rti obtained from l£A 
'occupatioiial education peAsonn6l at the tlme""bf the first and second 
on-site evaluations. In addition, data were obtained frbm the LEA 
occupational education personnel by a follow-up questionnaire sent to 
the personnel after the second on-site evaluation and from the on-site 
evaluation team me.mbers by a follow-up questionnaire. 

Selectioh and Description of the Sample LEAs 

I . ^: 

rhe fifteen LEA^ were selected to represent the various sizes and 
locations of LEAs within the, state of Illinois, two from within each of 
the seven lOE regions^ and one selected at large. 

Region and Enrollment of the LEAs in the Sample 

Six of the LEAs had a total enrollment at the secondary level of . 
less th^n 750 students, five had enrollments of between 751 and 1500 

f 

students while four had enrollments of more than 1500 students. These 

J 3 



> <|nrollraents provided a cross section of LEAs within the state of Illinois 
by LEA. size. For each region a median student enrollment was computed for 
those lEAs eligible for selection. The median student enrollment by region 

, varied from over 4500 stud^ts for Cook County to under 900. for region four. 

. •• > ' • . • • 

* Occupational Education Student Sample 

The total number of occupational education students from the fifteen 

selected LEAs - who responded to the Preliminary Evaluation Instruments - 

*• 

Student Form (PEI-S) at th^i time of the first on-site evaluation was 1387. 
Fo^ 'the second on-site evaluation the number of occupational education, 
students who responded to the PEI-S was 1867. * 

Grade Level of Students 

At the time of the first on-site evaluation, approximately 43% of 

the sample were classified as seniors with about 32?o classified as juniors, 

11% classified as sophomores ^nd 1% classified as freshmen. At the time 

of the second on-site evaluation almost half (49.8%) were classified as 

senior^ with approximately 267o classified as juniors, 18% classified as ^ 
s 

sophomores and 6% classified as freslimen. 
Occupational Area of Students 

At the timt' of the first on-site evaluation, 45.2?, of the occupational 

■ \ 

education scudunts who completed the PEI-S were c^assifiod in the business 
occupational ar^vi, 29*97, w^ui* classifiid in tilt industrial occupational 
arr-'a, 17.77', wori^ classified in tho liomt- economics occupational area, 4,7% 
WOT'- clissified in thi^ fujalLli occupaLions occupational area, and 2 • 57o w^ro 
rlassiiitu^ in a:;r icu i turi^ occupalionai areaT 



For tt^ second on-site evaluation, 36. 07, of the occupational education 
students were^cUssif ied in the business occupational area, 33.7% wera 
classified in the industrial occupational area, IS, 1% were classifieid in 
the home economics occupational area, 6, 3% were classified in the health 
occupations occupational area and 3,9% were classified in the agriculture 
occupational area. 

- H LE A Occupational Education Personnel Sample 

Data were obtained from LEA occupational education personnel at 
three different time?: 1) for the first on-site evaluation, 2) for the 
second on-;ite evaluation, and 3) by follow-up questionnaire. The total 
number of LEA occupational education personnel from the fifteen selected 

i 

LEAs who responded to the Preliminary Evaluation Instruipent - Personnel 
Form (PEI-P) at the time of the first on-site evaluation was 231 and for 
the sepond on-sit^ evaluation the total who responded to the PEI-P was 
408. From the two ort-site evaluations combined, a total of 639 PEI;P 
questionnaires were obtained for this study. The total number of LEA 
occupational education personnel from the fifteen selected LEAs who 
responded to the LEA Persorinel Follow-Up Questionnaire was 321, 

Personnel by Assignment at the I'irst ^and 
Second On-Site Evaluations 

At the time of the first on-site evaluation 69.3% of the occupational 
education personnel sample were, classified a^ having Ins'tv|;uct ion as their 
assignment^, 15. 2% classified as Guidance and 15.5% classified as 
Administration, At the time of the sei . on-site evaluation 68. 67o of 
the sample were classified as in Instruction with 14.07, classified as 



/ 
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Guidance and 17.4% classified as Administration. 
LEA Fersonnal by Assignment for the LEA 

Personnel Follow-Up Questionnaire • 

Of the 366 LEA occupational education personnel identified and 
selected to participa'wC in this study, 321 responded by completing the 
J^A Personnel follow-Up Questionmiire and returning iuby the end of the 
data collectidn period. For all fifteen selected LEAs combined the return 

\ 

rate was ■QS'L The rdwest rate of return for any one LEA was 74%; the 
highest 'rate of return was 100% which was accomplished at four LE^s; 

Of the sample that responded to the LEA Personnel Follow-Up 
Questionnaire, 47..4"/i^ere classified ^s Instruction (full time £n instruction), 
10.3% classified as Guidance (full time in guidance), 13.4% classified as. 
Administration (full time in administration), 10,9% as Other-non- ^ 
Administration (guidance, instruction, other), and » 18. 1% classified as 
Other-Administration (guidance and/or instruction and/or other and 
administration), ^ ^ 

* * 

Prior Experience of LEA Occupational 
Education Personnel on DAVTE On-Site 
Evaluation Teams 

Of those LEA personnel classified as Instruction 88*2% reported no 
prior experience on a DAVTE on-site evaluation team while 11. 8% reported 
prior experience* Only 9, VA of those personnel classified as Guidance 
reported prior experienc^e on a DAVTE on-site evaluation te'am; 41.9?. of 
those personnel classified as Administration reported prior experience; 
17, n of those personnel classified as Ot:her-non-Administration reported 
having pr^or' exper ience ; and,*36.2/'. of those'^personne 1 classified as 



Other -Administration reported prior experience on .a DAVTE on-site, . 
evalua'blon team. Thus, it was determined that those personnel with 

..3 ' 

administration as, -at least, a part of their assignment were more. /likely 

to have had prior experience on a DAVTE Qn-site 'evaluation than those 

petsonnel with -no .administrative assignment. ' ^ 

The LEA occupational edu(-;ation personnel sample has been grouped on 

the basis of assignment; and prior experience on a DAVTE on-site evaluation 

for several of the analyses in this study. Two of these sub-groups did 

not have sufficient numbers of personnel for -separate analysis and were 

not included; they were personnel classified as Guidance who had prior 

t 

experience on a DAVTE on-site evaluation team and personnel classified as 
Other-non-Administration who had prior experience on a DAVTE on-site 
evaluation- team« 

* • 

On-Site Evaluation Team Member Sample 

I 

/ • 

Of the 109 on-site evaluation team members identified and selected 
to participate in this study, 96 responded by completing' the Team Member 
Follow-Up Quest lonna ire and returning it by the end of the .data collection 
period. 

* 

The return rate was 88% for all fifteen on-site evaluation teams 
combined* The lowest rate of return for any one on-site evaluation team 
was 67%; the highest rate was 100%' which was accomplished for five teams* 

0£ the 96 team members who responded to the Team Member Follow-Up 
Questionnaire, 22.9% were employed in educational administration, 7.3% 
were employed In guidance, 46.9% were employed as instructors, 14.6% were 
employed tn business, labor or industry, and 8.3% were present or former 
occupationa 1 educat ion students. 



Summary of Findings for Research Question 1 

: ■ ■ . ♦ 

/ ■ 

\ . • 

• Pres.ehted in this section- is a summary of the findings for Research 
Question 1. The findings are presented by program component :- Planning 

t 

and* Evaluation, Student Services, Occupational' Programs , Personnel, 
Program Management, and Usi>__of Community Resources. 

' Research Question 1 was: ^* 
^ Have imp^ovements.^occurred in the occupational education programs 
from a fir^,t^^on-sit:e evaluation to a second similar on-site evaluation 
conducted fiv^ years later? ^ 

Discriminant analysis, analysis of variance (two-way) and Chi-square 
analysis were^ selected as the primary statistical tools for data analysis 
Discriminant analysis allows for the simultarieous analysis of multiple 
dependent and independent variables and aids in identifying which, if any 
dependent variable(s) contribute to significant differences, among the 
selected team member and LEA personnel sub-groups. 

Planning and Evaluation 

<^Guidance, Instruction and Administration personnel all reported 

a hiftjier level of involvement in this activity at: 'the time of the second 

on-site evaluation than they did at the first on-site evaluation. The 

combined lovel of involvement across'" all personnel was reported as being 

somewhat below moderatt^. LEA personnel were significantly more ^'nvolved 

in plauninj^ and/or makinp. changes in their districts' Ont; and Five Year 

Plan tor Occupational Education at the time of the second on-site 

t'valu.'Uion ilvxn at Lhe time of the tirsc on-site evaluation. Personnel 
« 

classiLiftl in Admini'stTa t i on reported a s igni i i cant 1 v lu^^^her level of 
Lnvolv»*mtnt tlian p»M*sonnel in Guidance and Instruction. 
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I 

Substantial improvc^ment in the development and use of stated and 
meiisurabU objectives was indicated by the comparison of the ETRs,for 
the first and second on-site evaluations. At the time of the first 
on-site evaluation only.. one. of the fifteen LEAs had stated and measurable 
objectives for their occupational education program^ At the time^of the 
second on-site evaluation the number of LEAs which had such objectives 
had increased to twelve. ^ 

the time of the first on-site evaluation only one of the fifteen 
LEAs had an operating- local evaluation system for the occupational 
education program. The number of LEAs that had such activities had increased 
to eight at the time , of ^the second visit.^ 

LEA personnel reijorted, via the LEA Personnel Follow-Up Questionnaire, 
improvement in 'the Planning arid Evaluation program component. Planning 
and Evaluation reci^ived the highest combined mean (2.86 on a scale of 1 * 
none, 2 » little, 3 » moderate, 4 * extensive) for extent of improvement 
across all eight selected LEA personnel sub-groups. This combined mean 
indicated that the eight LEA personnel sub-groups combined reporjted the 
extent of improvements in the Planning and Evaluation program component 
as being about moderate. ■ . . ^ 

Personnel in the Administ ration-N (no prii^r experience on a DAVIE on- 
site evaluation team), Adminis tration-Other-N, and Administration-Y (yes, 
prior experience on a DAVTE^-site evaluation team) sub-groups reported 
1 greater extent of improvement in the Planning and Evaluation program 
component than personnel in the other LEA personnel sub-groups.. Personnel 
in the Other-non-Administrat ion-N, Adminls trat ion-Other-Y sub-groups 

r 

roported a losser extent of improvement in the Planning and Evaluation 
pro^rvlm component than personnel in »the other LEA personnel sub-groups- 



The on-site evaluation team members reported via the Team Member 

» ■ < ■ ■ ' I 

'FollQW-Up Questionnaire improvement in the Planning and Evaluation 

«• ■ ' - ■• ' - 

program component. In relation to the team members reported exten^t of 

0 

improveinents for the ^her program components, Planning and Evaluation 
received the lowest reported extent of improvement. " The reported exteitfc 
...of improvement for Planning and Evaluati^^ was less than moderate but 
more than little (^.61). There was no sig^ficant differences in the 
repbrted extent of improvement in the Planning and Evaluation program 
component by the team member sub-groups • j 

Student Services 

OccupationaT education students rates, their guidance counselor's 
knowledge of the world of*,work somewhat ^above average but not high at 
the time of both Che first and second on-site evaluations. There was no 
significant difference found in the students' ratings of tl^ir guidance 
counselors' knowledge of the world of work f&r the first and the second- 



on-site evaluations. Industrial education students rated their guidance 
counselors knowledge of the world of work significantly lower than 
students in the other four program areas. " ' 

The occupational education students rated the information they had 
received frpm their guidance counselor with regard to theit future 
occupation as average at the time of both the first and iecond on-site 
evaluations. There was no significant difference found in the students' 
ratings of the information they had received from their guidance counselo 
with regard to their future occupation for the first and second on-site 
evaluations. 

LEA personnel ratod the guidance counselors' knowledge of 
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occupational course 'and program offerings abouc average (1.82 on a scale of 

r 

1 « ,high, 2*» average, 3 " low) at the time of the first on-site evaluation 
and well above-, average (1.68) at the time of the- second on-site evaluation. 
A sijjnificant difference was found in the LEA personnel's rating of the guidance 
coi+nsclors' • knowledge of occupational and course offerirtgs. for the first and 
second on-site evaluations. Guidance counselors were more knowledgeable abpUp 
occupational course and program of<ferljn^ at the time of the s|Cond^on- 
site evaluation than at the time of the f^irst on-site evaluation. 
'•Occupational instructors rated the guidance counselors* knowledge of 
occupational course and program offerings significantly lower than did 
the other LEA' personnel.^ * * * ^ 

Very little improvement in placement servi,ces was indicated by 

\ 

the comparison of the ETRs forN;|ie 3Si^^^ second on-site evaluations. 

Some improvement was indicated in the LEAs testing of career . interests 

as was indicated in the availability of more career -information. \ 

t * • 

\ a 

LEA personnel ^reported improvement in the Student Services program 
component. The Student Services program component received 'the fourth 
highest combined mean (2.72) for extent of improvement across .all^eight 
LEA pers'onnel sub-groups. This combined mean indicated that 'the eight 
LEA personnel sub-groups combined reodrtad the extent u: improvements 
•in Che Student Services program component as being less than moderate 
but more than little. 

Personnel in the Other-non-Adminis tration-N and Administration- 
Other-Y sub-groups reported a greater extent of improvement in the 

Student Services program component than personnel in the othnr LEA 

* 

personnel sub-groups. Personnel in the Administratton-N, Adminis t ration*""*"^ 
Ocher-N, and Adminis tration-Y sub-groups reported a lesser extent of 

2i 



improvement in the Student Services program component than personnel in the 
other LEA personnel subrgroups. « ♦ , |^ ' 

The on-site .evaluation team members also reported improvement in the 
/^Student Services program component. The Student Services program component 
received the fifth highest combined mean (2.77) 2or exn:ent of improvement across 
all five team member sub-groups. This combined meaii indicated that the five 
team member sub-groups combined reported' the extent of improvements in f the - • 
Student Services program component a^ being less than moderate but more than 
little. There was.iwo signifxcapt differences in .t;he reported extent of improve* 
mept in the Student Services program component by the team member sub-groups.. 



.. .Occupational Ifro^rams 



Occupational education students* rated their occupational program in 
terms of preparation for a career somewhat above' average but not high at the 
time of the first and second on-site evaluations.", The combined mean rating 



for the" second -on-site visit was higher '.han for the first on-site visit. 

There was a significant difference in the means of the ra'-..ags among 
occupational areas#- Students in th^ home economics program area gave a 
significantly lot^yer rating o£- their program's ability to prepare them for' 
an occupation than students in the other program areas. 

A significant c^rop in the percentage Of occupational education 
students participating in stxident vocational clubs was found. With ^the 
exception of the agriculture area., all student organizations wer^ found 
to have had substantial drops in their student membership. The student 
vocational clubs in the agriculture area wiro found to be increasing in 
their percentage of sLudont membership. 

Improvements were noted in the second on-site evaluacion ETRs for 



sequencing occupational courses into structured i^programs and in the offering 

of programs in each o,f the five occupational 'areas. Very little inSprovement,, 

if any, was noted in t^ip^ support for occupational student org'anizationSf 
» • * 

There has been a substantial increase in the number of ;con*c^usibns 

contained in the occupational programs .section sof the ETRs^from the time 

of the first on-site evaluation (85.Lto the second- on-site evaluation (295). 

LEA personnel reported improvement in thrf Occujpational Programs 

program component. The Occupational Program.s program componenc received 

the second highest combined mean (2. 85) for- extent of improvement across- . 

'all eight LEA personnei sub-groups. This combined mean indicated that 

the eight LE^ personnel sub-groups combined reported the Extent- of 

improvenwuts in the Occupational Programs program component as being about 

/ ' 

/ moderate. • * . • ^ 

Personnel in the Administration-N, Administration-Other-^N, and 
Administ;ration-Y reported a greater "extent of improvement i^i the Occupational 
• Programs program* component than personnel in, the other LE^ personnel sub- 
groups. Personnel in the Other -non-Administration-N and Administratioh- 
♦ Other -Y reported a lesser extent of improvement in th^^ccupat ional 
Programs program component than 'personnel in the other LEA perspnnel . 
sub-groups. 

The on-s .te evaluation team members aLsc. Reported improvement in 
the Occupational Programs program component. The Occupational Programs 
program component received the highest combined mean (2,96) for 6xtent of 
Improvements across all five te^ member sub-groups. This combined mean 
indicated that the five team member subrgroups combined reported the 
extent of improvements in the Occupational Progi^aras'program component 
as being moderac>i» Tht^re was no significant difference in the reported 



extent of improvemenss in the Occupational Programs program' component by 
the team member sub-groups. 
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Occupational educati,6n, students rat^fd theixj; occupational instructors* 
knowledge of the world of work well above/average but not high. The . 
occupational education a.|:udents rate4 their^^ccupAtioiial instructors'^ knowledge 
"Qi Che world -of work significantly iiigher at the- time of .the .second on-site 
visit than at" the time of the first on-site visit. There was also a signi- 
ficant difference iji the means of the' ratings among, occupational areas. 
Stude^fi^s in the health opcupations program area- gave a" significantly higher 
rating and students in the home' economics program area gave a significantly 
• lower racing of their ins tr.uc tors.', knowledge of the world of work than students 
in the other program areas. ' ^ - ' \ 

*LEA personnel rated the working relationship between occ\>pa tional* and 
acadenlc instructors somewhat above average but not high. The LEA personnel 
rated the working relationship between occupational- and academic instructors 
significantly higher at the time of the second on-§ite visit than at the time 
of the first on-site visit. There was also a significant difference' in the 
means of the, ratings among personnel assignments-.' Adipinis trators rated the 
working relationship significantly higher than did^guidance or instructional' 
personnel . . . 

LEA personnel rated. the working relationship between guidance and ins true- 
tional personnel at *the time of the;- first and socond on-sito. visi'ts. There 
wa.s a siguificant differ-once in- the means of tho ratings^among personnel * 
assignnu»n ts . Occupational instructors rated tho working . relationship 
significantly lower , than did H^i^^nco or administrative personnel. 



Improvement in the occupational experience of the occupational education 
personnel was .indicated by^^e comi/arison of the ETRs for the firsthand 
second on-site evaluations. Improvement was also indicated in the 
/ssignment of occupational t^ducatioa personnel to teach in areas for 
which they .were qualified. . ^ ^\ 

LEA personnel reported improvement in the Personnel program .component 
The Personnel •program component received the Lowest combined mean (2.57) 
for extent o£|^provemenC. across all eigjic LEA personnel sub-groups.' This 
..combined mean, indicated chat Che eit,nt LEA personnel sub-groups combined 
reported Che extent 6f improvements in the Personnel program component as 
being about half-way , between moderate and little. ' " . ^ 

. iPersonnel in the Administration-N, and Administration-Other'N, and 
Administration-Y sub-groups reported a greate'r extent of improvement in 
*th'e, Personnel program 'comp'onent than personnel in the other LEA personnel 
sub-^groups. Personnel in the Other-non-Admintstration-N and Administration-^ 

Other-Y reported a lesser extent of irnqprovements in the Personnel program 

. / • ■ . 

component than personnel in the other LEA personnel s^b-groups. 

The on-site evaluation , team members also reported improvement in. the 

. personnel program component. The Personnel prC^ram component received the 4 

fifth highest combined mean (2.77, - tied with Student Services) for extent 

of improvements across -all five team member sub-groups. This combined mean 

indicated chat the five sub-groups combined reported the extent of 

improvoment in the Personnel program component as buing somewhat less than 

modorate but more than little. There was no significant diffurencc in the ^ 

reported extent of improvement in the Porsonnt^l progranv component by the 

t. am member sub-groups. 
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LEA personnel rated their loca.l board of education's support for 
occupational education average (1.96) at the time of the first oti-site 
visit and somewhat above average (1.79) at the time of the second cJn-site . 
visit. The LlilA personnel rated the boards* support for occupational education 
significantly higher at the timtf of ^the second on-site visit than at the ' 
time of the first on-site visit. There was also a. significant difference 
in^ the means of the rating among personnel assignments. Administrators rated 
the boards' support for occupational education signif icantly^higher than did 
guidance an^'i instructional personnel. 

LM-^^i^sonnel rated 'the working relationship be tween^occupa tional 
instructors across all occupational areas well above average but not Jiigh^ 

... . . ■ • 

at the time of the, fix*:- 1 (1.68) and sSecond (1.60) on-site visit^^. There t 
was no significant difference between the ratings of tlie working relationships 
between occupational instructors -across all occupational areas at thfe time 
01 the first and second on-site visits. There was also no significant 
difference in the means pf the ratings among personnel assignments. 

LEA personnel rated the'.working relationship bfstween occupational 
instructors and occupational program administrators well above average 
but not high at the time of thfe first (1.64) "and second. (1.58) on-site 
visits. There was no significant difference between the ratings of the 
workinj^ if la t ionsh Ip between occupational instructors and occupational 
program administrators at the time of the first and -^second on-site visits. 
There was a i^igniiicant difference in the means of the ratings among 
personnel a*?stKninents* Administrators rated the workin'g relationships 
s i ■.:ni L Leant ly h L*;lu:j?'^thani did guidance and instructional pc^i*sonnol. 
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Imprqvement In the support for occupational education by local boards 
of oducatiW^as indicated by the comparison of the ETRs for the first and 
second on*site evaluations. Improvement was also indicated in vertical 
articulation and the use of written job descriptions* 

LEA personnel reported improvement in the Program Management program 
component. The Program Management program component received the fourth 
highest combi):ied mean (2.67) for extent of improvement across all eight 

> 

/ 

LEA personnel sub*groups. This combined mean indicated that the (2ight 

LEA personnel sub**groups combin^id reported the extent of improvements in 

the Program Management program component as being somewhat less than 

moderate but more than little. 

J Personnel in the Other-non-Adminls tration^N and Administration-Bother -Y " 

reported a greater extent of improvement in the Program Management pcpgram 

component Chan personnel in the other LEA personnel sub-groups . Personnel 

in the Adminis tration-N, Administrat ion-Other-N, and Administration-Y 

sub-groups reported a lesser extent .o£ improvement in the Program Management 

program component than personnel in the other LEA personnel sub-groups. 

The on-site evaluation team members also reported improvement * in the 

Program Management T^rogram component. The Program Management program 

component received the third highest combined mean (2. 80) for extent of 

improvements across all five team member sub-groups. This combined mean 

indicated that the I'ive team member sub-groups combined reported the extent 
» 

or improvement in the Program yanagoment program component as being a littlu 
Kiss tluin moderate. There was no significant diftertinco in thi! reported 
ox tent of improvements in the Program Management program componv^ncs by tht 
team member sub-groups. 




Use of Congnunitv Resources 



LEA personnel rated the working relationship between occupational 

• V 

instructors and labor and management personnel in government^ business, 
trade and commerce as being somewhat above average at the time of the 
first (1.81) and second (1.83) on-site visits. There was nonsignificant 
^difference found in the LEA personnel's rating of thi^ working relation- 
ship for the first and second on^sita evaluations. There was also ;io 
significant difference among the meaas of the ratings by LEA personnel 
assignment. 

Improvement in the establishment and use of advisory committees was 
indicated by Che comparison of the ETRs for the first and second on-site 
evaluations. % 

• > V 

( 

LEA personnel reported improvement in the Use of Comnfunity Resources 
program component. The Use of Community Resources program component 
received the third highest combined mean (2, 76) for extent of improvement 
across all eight LEA personnel subrgroups. This combined mean indicated 
that the eight LEA personnel sub-groups combined reported the extent of 
improvement in the Use of Community Resources as being somewhat less than 
moderate. 

Personnel in the Adminis tra tion-'N, Adminis trat ion-Other-N, and 
Administracion-Y reported a greater extent of improvement in the Use of 
Contmunity Resources program component than personniel in the other LEA 
personnel sub-groups. Personnel in the Or.her-non-Adminis trat ion-N and 
Adminis tratiion-Other-Y reported a lesser extent of improvement in the 
rst^ ot CoTTiTmnity Resources program component than personnel in the other 
LEA personnel sub-groups. 
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The on-site e^^aluation team members also reported improvement in the 
Use of Community Resources program component. The Use of Community Resources 
program component received the second highest combined mean (2.87) for, 
extent of improvement across all five team member sub-groups* This combined 
mean indicated that the five team member sub-groups combined reported the 
extent of improvement in the Use of Community Resources as being about moderate • 
There was no significant differences in the reported extent of improvement 
in the Usf of Community Resources program component by the team member 
sub-groups* 

\^ 

^Summary of Findings for Research Questi^ 2 

Presented in this section is a s^immary of the findings for Research 
Ques^tion 2. The findings are presented by their source: LEA Personnel 
Follow-Up Questionnaire and Team Member Follow-Up Questionnaire. 

Research Question 2 was: 

Was there a relationship between improvements in the occupational 

education program and the on-site evaluation? 

» 

•J 

The Relationship Between Improvements in the 
Occupational Education Program and the 
On-site Evaluation as Reported by LEA Personnel 

The LEA personnel reporced that the planning and evaluation program 
component had the highest combined moan (2.48) for the extent of relationship 
hftween improvomt-nt in tlw occupational program and the first DAVTE on-site 
-•valuation ;icrosa. all <'ight LEA personnt'l sub-groups. This combined mean 
indicatrd that tht- combtn.-d r.ight lEA personnel sub-groups raced the extent 
.It relationship between improvement in the Planning and Evaluation program 



^ofRi>orvpnr, and the DA VIE on-site evaluation littla to moderate. 

The second highest combined mean (2.42) for the extent of relationship 
between improvement and the DAVTE on-site evaluation was reported for the 
Occupational Program program component. In order, the other program 
components were Program Management with a combined mean of 2.38, Use of 
Community Resources with a combined mean of 2»37, Student Services with a 
combined mean of 2.77, and Personnel with a combined mean of 2.17. The 
reported extent of relationship between improvements and the DAVTE on-site 
evaluation in the six program components ranged from (2.48) little to 
moder^e to (2.17) little. 

The LEA personnel 9ub-group reporting the highest combined mean (2^77) 
extent of relationship between improvement and the DAVTE on-site evaluation 
across all six program components was reported by those persons classified 
as Administration-N. The second highest combined mean (2.64) for extent of 
relationship between improvement and the DAVTE on-site evaluation across all 
six program components was reported by persons classified as Ins truction-Y« 
The other LEA personnel sub-groups, in order of combined means were: 
Administracion-Other-N (2.55), Adminis tr-^tion-Other-Y (2.40), Ins truction-N 
(2.28), Administration-Y (2.21), Guidance-N (2. II) and Other-non-Adminis tra- 
tion-N (2,00). The reported extent of relationship between improvement "and 
the DAVTE on-site evaluation across all six program components ranged from 
(2,77) about moderate to (2.00) Iitt;le. 

One discriminant function was significant at the .05 level. When one 
or more discriminant functions was found to be significant, this indicated, 
in tliis particular analysis, that a significant difftirunce existed in the 
roporti^'d i?xtent of rt^ationship between Improvevaent in one or more of the 
six program components and the DAVTE on^sLte evaluation by at least one of 
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the selected LEA personnel sub-groups. 

The ^ight LEA personnel 8ub--groups became arranged into three clusters: 

1) Administration-Y, Other-non-Admlnistratlon-ha and Adn.4.nlstration-Other*N; 

2) Instruction'-Yy and Administration*W; and In8truction-*N, Guldance-N, and 

3) Adminlstration-Other-Y. The cluster comprised o£: Admini5tratlon7Y» 

0ther--non*Adinini5tration--N» and Admini5tration-0ther"-N reported a greater 

« 

extent of relationship between improvement in the program component Use o£ 
Community Resources and the DAVTE on-site evaluation, and a lesser extent 
« .o£ relationship between improvement in the program components o£ Personnel 
and Student Services and the DAVTE on-site evaluation. The cluster 
comprised of Administration-N and Instruction-Y reported a greater extent of 
relationship between improvement in the program components of Personnel 
and Student Services and the DAVTE on-site evaluation and a' lesser extent 
of relationship between improvement in the program component Use of Community 
'Resources and the DAVTE on-site evaluation. The cluster comprised of 
Instruction-N, (guidance -N and Administration-Other-Y reported relatively 
small differences'^in the extent of relationship between improvement among the 
six program components and tUc DAVTE on-site evaluation. v 

The Relationship ^qtween Improvements in 
the Occupational Education Program and 
the On-site Eva.aatiou as Reported by 
On-site Evaluation Team Members 

The on-site evaluation team members reported that the Occupational 
Programs program component had the higheSu combined mean (2.89) for the 
extent of relationship between impiovement and the DAVTE on-site evaluation 
across all five team monbcr groups. This combined mean indicated about a 
moderate level ot re lationship. 



The second highest com)3ined mean (2»81) for extent of relationship 
between improvement and ^jtho DAVTE on^sUe evaluation was reported for the 
Use of Community Resources program component*. In order, 4:hei other program 
components were Planning and Evaluation with a combined mean of 2.70, Program 
Management with a combined mean of 2*68, Student Services with a combined 
mean of 2.64^ and Personnel with a combined mean of 2.55* The highest reported 
extent of relationship between improvement and the DAVTE on-site evaluation 
in the six program components was reportea by persons classified as Guidance. 
The second highest combined mean (2.74) extent of relationship between 
improvement and the DAVTE on-site evaluation across all six^program components 
was reported by persons classified as Administration. The. other team - 
member groups in order of combined means were; Instruction (2.73), Busineg^s 
Industry and Labor Personnel (2.69), and Occupational Students (2.33). The 
reported extent^^^ relationship between improvements and the DAVTE on--aice 
evaluation in the six program components by each of the five team member 
groups ranged from (2.83) about moderate to (2.33) somewhat above little. 

There was no significant discriminant function as a result of the 
discriminant analysis. Therefore, no significant differences existed in • 
the reported extent of relationship between improvements and the DAVTE on-site 
evaluation among' the six program components and the five team member sub-groups. 

Summary of Findings for Research Question 3 

. Presented in this section is a summary of the findings for Research 
Question 3 and its subques tions . The findings are presented by their source 
of data: 1) LEA Personnel Follow-Up Questionnaire, and 2) Team Member 
Fol low-Up Questionnaire. 

Research Question 3 was: 
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Did the LEA occupational education personnel and the members of the 
9n*slte evaluation teams see the on-site evaluation as being of value? 

The Value of the On-Site Evaluation" 
as Reported by ^ LEA Personnel 

LEA .personnel reported that the on-site evaluation would help them 
to Improve their occupational education program^ Over 69% of the 
respondents reported that the on-site evaluation was beneficial to their 
occupational education program. There was no significant difference found 
between the value of the DAVTE on-site evaluation and LEA personnel assign- 
ment. 

LEA personnel also reported that the on-site evaluation was not a waste 
of time and money in terms of its impact on their occupational education 
program* Two-thirds of the respondents reported that the on-site evaluation 
was not a waste of time and money. There was no. significant difference 
found between the impact value of the DAVTE on-site evaluation and LEA 

personnel assignment. 

.t 

LEA Personnel Perceived Value of Activities 
Related to the On-Site Evaluation 

LEA personnel reported that the orientation meeting for the on-site 
evaluation provided them with an adequate understanding of the on-site 
evaluation process. Over 807o of the respondents reported that the 
orientation meeting provided them with an adequate understanding of the 
on-site evaluation process. LEA personnel with an administrative assignment 
reported a significantly higher level of value for the orientation meeting 
in providing an understanding of the DAVTE on-site evaluation process, while 
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those persons with a guidance assignment reported the lowest level of value. 

LEA personnel reported that the on-site evaluation lielped them to 
become much more familiar with their district's One and Five Year Plan fot 
Occupational Education than they were before the on-site visit. Over half 
of the respondents reported that they wer ^ much more familiar with their 
disti:icL*s One and Five Year Flan for Occupational Education as a result 
0% the on-site visit. LEA personnel with an instructional or administrative 
assignment gained a higher level of familiarity while persons with a guidance 
assignment reported the lowest level of gaining familiarity. 

LEA personnel also reported that the activities they "X^ere involved m 
while preparing for the on-site evaluation were beneficial. Almost half of 
the respondents reported that the activities they were involved in while 
preparing for the DAVIE on-site evaluation were beneficial to themselves. 
LEA personnel with an administrative assignment reported a significantly 
higher level of benefit while persons with a guidance assigxunent reported 
a lower level of benefit from the activities. 

Involvement of LEA Personnel in Preparing 
for the On-Site Evaluation 

LEA personnel reported their level of involvement in preparing for the 
on*site evaluation for ten selected activities • 

The faculty meetings activity received the highest combined mean (2.93) 
for level of involvement across all eight selected VLk personnel sub-groups. 
This combined mean indicated that the eight LEIA personnel sub-groups combined 
reported their It^vel of involvement in faculty meetings as being moderate. 

r The second highest combined mean (2.90) for level of involvem^-rnt was 
reported for the activity of discussions with ocher faculty members. In 

■? 
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order I the other activities were: reviewing occupational course objectives 

with a combined mean o£ 2.67, conducting self-evaluauion activities with a 

combined mean of 2.66, reviewing occupational program goals with a combined 

mean of 2«S9| discussions with occupational education admlni8.trators with a 
* * 

combined mean of Z.SS^ reviewing Local District One and Five Year Flan with 
a combined mean of 2.49, discussions with guidance .personnel with a combined 
mean of 2.32 ^ meeting with an advisory committee with a combined mean of 
2.32f and meeting wit^h occupational student organizations with a combined 
mean of 1.69. The reported level of involvement in the ten selected 
actiyj^ties ranged from (2.93) moderate to (1.69) less than little. 

The LEA persoiyiel^ sub*group that reported the highest combined mean 
(2.84) level of involvement across all ten selected activities were comprised 
of those persons classified as Administration*Other with experience on a 
DAVTE '^on-site evaluation team. The other LEA personnel sub-groups in order 
of combined means were: Administration with experience on a DAVTE on-site 
evaluation team (2.61) » Instruct;ion with experience . on a DAVTE on-site 
evaluation team (2.58) » Administration with no experience on a DAVTE on-site 
evaluation team (2.52), Administration-Other with no experience on a DAVTE 
on-site evaluation team (2.38), Other-non-Adminis tration with no experience 
on a DAVTE on-site evaluation team (1.87). The reported level of involvement 
across all ten selected activities by each of the eight LEA personnel sub- 
groups ranged from (2.84) about moderate to (1.87) about little. 

LEA personnel sub-groups with experience on a DAVTE on-site evaluation 
team reported a higher level of involvement across all ten activities than 
did the other LEA personnel sub-groups with no experience on a DAVTE on-site 
evaluation team* 

Two discriminant functions wore significant at the .05 level. Since 
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two discrimiiiant functions were, found to be signlflqant, this indicated t ;at 
a significant «Ufference iij the reported levels of involvement in tlje ter 
selected activities by the LEA per;sonnel sub-groupiSt • ' ^ 
' 'With respect to the^ first discriminant function, the eight LEA 
personnel sub-groups were arranged into three clusters; 1) Other-non- 
Administration-N, Ins truction-Nj and Administration-Other-Y; ^2) *Adminis tra- - 
\ tion-Oth<ir-N, Adminis tration-Y, I^structionW, and Administration-N; and 
3) Guidance-N. The cluster comprised of: Other-non-Administra tion-N, 
Instruction-N, and. Adminis tra tion-Other-Y reported a greater level of 
* involvement in discussions with other faculty members and reviewing 
occupational course objectives and a lesser involvement in discussions 
with' guidance personnel. The cluster comprised of Guidance*N reported a 
greater level of involvement in discussions with guidance personnel and a 
lesser level of involvement in discussions with other faculty members and . 
reviewing occupational course objectives. The cluster* comprised of; 
Administration-Other-N, Administration-Y, Ins trucCion-Y* and Administra tion-N 
reported relatively -small differences in their level of involvement among the 
ten selected activities. 

With, respect to the second discriminant function, it was concluded that 
the eight LEA personnel sub-groups were arranged into three clusters: 
I) Adminis tration-Y, and Adrainis tration-Other-Y; 2) Instruction-N, Guidance- 

Ins true tion-Y, and Other-non-Adminis tra tion-N; and 3) Adminis tration-N, 
and Administration-Other-N. The cluster comprised of Adminis tratlon-Y and 
Adminis tra tion-Other-Y reported a greater level of involvement in meeting 
with advisory committees, reviewing Local District One and Five Year Plans, 
and discussions with guidance personnel and a lesser involvement in 
discussions with other faculty members and reviewing occupational course 
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objectives. The cluster comprised o£ Ins true tion-N^ Guidance-*N) Instruction* 
Y, «nd Other-non-Administration-N reported a greater level of involvement in 
discussions with other faculty memlers and reviewing, occupational course 
objectives and a< lesser involvement in cneeting with advis.>r;/ coimpittees, 
reviewing Local District One and Five Year Plans, and discussions with 
guidance .personnel • The cluster comprised of Admit^istration-N and 
Administration-Other-N reported relatively small differences in their 
vlevel of involvement among the ten selected activities,. 

The Value of the On'Site Evaluation as 

Reported, by bn-Site Evaluation * 
Team Members 



r 

On-site evaluation team members reported that the on-si^^valuation 
would have a positive effect on the district's occupational educ|ftion 
program. Almost 90% of the respondents reported that the on-site evaluation 
would have a positive effect on the district's occupational education 

program. On-site evaluation team members classified as Guidance repor>ted 

C> 

^hat the DAVTE on-site evaluation will have a Iqss positive effect on. ehp 
district's occupational education program than the other members of the 
on-site evaluation teams t 

The on-site evaluation team members reported that the on-site 
evaluation was not a waste of time and money in terms of its impact on 
improving the district's occupational education program. Almost 907o of 
tho respondents reported that 'the on-site evaluation w^s worth iiheir time 
and etfort. There was no significant difference between the perc^^ived 
attitude ot the LEA personnel toward the DAVTE on-site evaluation and team 
member sub-groups. ' 

■d7 
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Tiiam Mfciinber Ruported Valye of the DAVT|I On-Sitc * , 
Evaluation to the Members of the DAVTE " • 

On-site Evaluation Team . . . ^ ' 

On-site evaluation team members reported that the experience obtained 
as a member of an on-site evaluation team would be valuable to anyone 
concerned with occupational education. All (10070 respondents reported 

that the experience was a valuable experience. There was no ^significant 
difference found between the value of the DAVTE oK-site evaluation , experience 
by team member sub-groups. 

The on-site evaluation team members also reported tha.t the on-site 
evaluation team ^experience had encouraged them to work further for the 
improvement of occupational education in their own district. Over 9S7» of 

\ ... 

the respondents reported that the experience had encouraged them to help 
improve their own occupa^inal education program. There was no significant • 
difference found between the level of encouragement and team member 
sv4b-groups. 

The on-site evaluation team members reported that during their inter- 

* 

views with the LEA personnel they got some good ideas on how to improve 
their own occupational education programs. Over 857o of the respondents 
reported that they had received some good ideas for improving their own 
occupational education program from the LE^ personnel thgit they interviewed. 
There was no significant difference found between the extent of obtaining 
j»ood ideas for improving their occupational education program and toam 
member sub-groups. ) 
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Discussion 



This atxxdy has Identified several aspects of the occupational education 
programs in Illinois that have been Improved since the time of the first 
on^'site evaluation^ There has been differential leve|,s of improveipent by 
the various program components, Also^ ' in- some cases, the level of improve* 
ment for some o^f the program components was reported to be different by the 
various reference groups providing evaluative ratings. 

The on**site evaluation teams at the time of the first on*site evaluation 
attempted to rate the quality of the 8 Areas of Concern by the use of an ^ 
absolute rating scale based on the ''ability" or resources^ of the LEA. 
Suxmnary ra,tings for the LEA were found to be confusing *and often counter** 
productive and, therefore, were discontinued; no further attempts to develop^ 
an absolute measure of program quality for LEAs or the program components ^ 
were made. 

No program quality index existed for use and this study did not 
attempt to develop such an Index. ' The measures used in this study were 

those reported by phe individuals composii;ig the various groups which 

• • • 

gave their perceptions of the, attribute being rated. Therefore, the 
membtsrs of the different groups very likely rated the same attributte 
differently; each relative to their own perception of the valpe or 
extent* of the attribute they perceived to :xist. 

'Although the reported levels of , improvement were not based on an 
absolute index of^ program quality, their use in this study does provide 
an adequate basis for determining program improvement. The on-site 
twaluation phase of the TPS was developed to assist the LEAs to improve 
their occupational education programs, whether that involved taking a 
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poor program and helping it become an average program or taking a good ' 

' . . .. " •. , \ • ■ • ' 

program and helping it become an excellent program. While <^ccupational 
education students^ LEA. personnel and team members may have reported 
different relative^, leva Is, of improvement based on their own perceptions » v 



the primary concern for this study was whether or not these groups found ' 

. • * V 

I I ' ' * 

; • 
improvement* . ^ • ^ 

' . •. ■ : .,; 

. This section relates the ratings* o^.LEA programs and rating? of ' 

improvements to the DAVTE on-site evaluation and the literature related 

.ft' 

to progii'am evaluation* , - ^ 

There were several significant improvements reported for the Planning 
and. Evaluation program component; this program compon^nt> included objectives 
,and evaluation which were separate Areas of Concern at the first on-^te 
evaluation. LEA occupational education personnel reported t^ at they were 
more inyolved in planning their LEA^s Qne and Five Year Plan for Occupatipnal 
Education chan/chey were at the time o'f the flrsst eValuatlon. Bowling (1973) 
found in his study of the Illinois TPS th^ft LEAs which had been eve^luatid. by 
an on-site evaluation team were more likely to involve more of the LEA, 
personnel in developing the One and Five Year Plan than were involved in 
urievaluatQd LEAs, He also found that LEA personnel with an administrative 
assignment were significantly more involved in this activity than were 
any other LEA personnel,. This study had similar findings: personnel . 
with administrative assignments were more involved in all aspects, and 
persons who had served on on-site evaluation teams %ere more involved at 
cheir own LEA •that were those who had not served on a team. 

The^ ETRs for the second on7site evaluation indicated tha< substantial 
improvement had been made in the use of stated and' measurable objectives 
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and in the use of local evaluation activities by the LEAs. Bowling (1973) 
Stated, "The evaluation teams (first on**site evaluation) were most concerned 
about the areas of objectives and evaluation (p. 136).'' As a result of 
this concern, the DAVTE undertook a statewide pt^ogram to^assist LEA 
personnel in developing stated and measurable objectives and also developed 
the Locally Directed Evaluation program to assist the LEAs in conducting 
their own evaluations for program improvement. The emphasis placed on these 
two aspects of the Planning and Evaluation program component by the first 
cn-site evaluation teams and DAVTE is reflecte-" In the significant improvement 
found in this area by the study. 

• The LEA personnel rated the level of improvement for the Planning and 
Evaluation program component higher than for any other program component. 
The school personnel who were at the .LEA at the time of the first on-site 
evaluation were probably involved in developing program and course objectives 
as well as being involved in evaluation activities and therefore aware of 
the changes at their LEA from the first to the second team visit. 

A somewhat different rating was reported by ttie on-site evaluation 
team members; they rated the level of improvement for the Planning and 
Evaluation program component lower than for any other program component. 
These seemingly contradictory findings may be the result of several factors. 
The first is that it is difficult to evaluate (as an outsider) change over 
time when the rater is using a single point in time *as a reference. 
Secondly, the other five program components are more easily identified 
with people and/or places whereas Planning and Evaluation is somewhat 
more abstract. It is also possible that the team members had a higher 
expectancy £or this program component dua to the widespread concern for this 
area ov^r a period of time. They may have been using the DAVTE, LDE materials 
as the expected standard. 
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The Student Services program component had very little reported 

Improvement from the time of the first on-site evaluation* Occupational 

education students reported no improvement in the guidance personnel's 

knowledge of the world cf work oy^ifTlihe information they received frqm 

> their guidance counselor relative -to informatjfdn about their future career. 

These areas wore rated average by the occupational education students at the 

time of both on-site evaluations, industrial oriented students rated the 

guidance counselors' knowledge of the world of work significantly lower 

than did the other occupational education students. 

Bowl 'tig (i973) indicated that most of the early char|g6s lmplemen;:ed in 
the guidance services area were in the rea of placement services t This 
early change was somewhat substantiated by the improvement found in the placer 
ment services as a result of the analysis of the ETRs. The ETRs also Indicated 
that more career infomation was available to occupational Education students 
at the time of Lhe second visit. 

The LEA personnel rated the level of Improvement for the Studenf . 
Services program component fourth relative to the improvement in the 
six program components. The comparatively little Improvement reported for 
this program component by occupational education students was similar 
to the findings from analysis of tl. ETRs and LEA personnel ratings. The on- 
site evaluation team members rated the level of improvement for the Student 
Services program component fifth. Therefore, improvement in the Student 
Services program component area was rated fifth on an over all basis. 

The Occupational Programs program component was found to have 
received differential levels of reported improvement from the time of the 
first on-si te- eva luatlon . Occupatlpnal education students reported no 
difference, no improvement, in the value of their programs in preparing 
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chem for an occupation. It was interesting to note that students. In home 
economics rated their program significantly lofwer in terms of preparation 
for an occupation than did the students in the other occupational education 
areas • 

One major concern was found in student membership in vocational clubs. 
A very significant drop in the percent of students who reported membership 
was reporS:ed by the occupational education students in all. areas except in 

the agriculture area. There was little mention of student membership by 

• \, 

the ETRs for the first on-site evaluation (Bowling, 1973) or for the second 
visit. 

An improvement in the sequencing of courses into programs was indicated 
by the'^analysis of the EXRs. Also, mor.^ LEAs were offering programs in all 
five program areas than were doing so at the time of the first on-site 
evaluation. Bowling (1973) found that the major area of change made by 
evaluated LEAs (first on-site evaluation) in the Occupational Programs 
program component was in the renamin^^^of courses and minor course changes 
which allowi:d for the sequencing of courses into programs^. * 

A major finding in the Occupational Programs program component was the 
substantial increase in the number of conclusions made by the second on-site 
evaluation team reports in^comparison to the first on-site evaluation team 
reports; totals for the 15 LEAs were 85 conclusions for the first on-site 
evaluation and 295 conclusions for the second on-site evaluation. Also, 
tnc nature of the conclusions was different. The ETRs for the second on-site 
evaluation contained conclusions concerning individual program areas in 
addition to general conclusions for the total Occupational Programs component 
whereas the first team reports reported only the latter. LEA personnel 
from guidance and the individual occupational areas expressed concern 
about not having a member on the team from their area. 
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This increase in emphasis on the individual occupational program areas 
within the Occupational,^ Programs component may indicate a need to reevaluate 
the method of selecting on*site evaluation team members • As structured 
some teams did not have a knowledgeable representative for each program 
area or for guidance. 

Th6 LEA personnel rated the level of improvemant for their Occupational 

Programs program component second relative to the six program components. 

As most of the LEA personnel were actively involved in the occupational 

programs it would seem likely that they would be in a position to be know ' 

ledgeable of any improvements that had taken place. The on-site evaluation 

team members rated the level of improvement, for the Occupational Programs 

program component higher than for any other program component. Several 

factors may be involved in this finding. As stated earlier a substantial 

increase in the emphasis placed on the. Occupational Programs program 

component by the on-site evaluation teams was found. This change in 

« 

emphasis may have caused the team members to spend more of their time 
evaluating this program component, thus finding more improvements. 

As with the Occupational Programs program component, differential 
levels of improvement were reported for the Personnel program component. 
Occupational education students reported improvement in their occupational 
instructors* knowledge of the world of work. It was interesting to note 
that students in the health occupations program area rated their instructors' 
knowledge of the world of work significantly higher than the students in 
each of the other program area^. This finding should not be too' surprising 
since the typical background of health occupations instructors involves 
somu type ot current certification and experience in one or more of the 
health fields along with, or followed by, the professional oducatl»on 




courses necessary to teach. Further, these programs typically Involve an 

■ *: 

operational setting, clinical experience*, Home economics students rated 
their instructors* knowledge o£ the world of work significantly lower than 
the students in each of the other program areas; this finding is consistent 
with the significantly lower rating given by the students or] these programs 
in terms of preparation for an occupation. ( 

LEA personnel reported an improvement in the working relationship 
between occupational and academic instructors but reported nol change in 
the working relationship between occtipa tional instructors and guidance 
personnel. Bowling (1973) reported that most LEAs were planning In-school 
workshops as a part of their personnel development program. These 
workshops were to include instructors and guidance personnel. It is 
interesting that an improvement was reported in the academic/occupational 
instructor working relationship but not for the working relationships with 
guidance personnerl. Guidance personnel were less involved in all aspects of 
the cccii^ational education program than other LEA personnel giupUps. 

LEA personnel rated the level of improvement for the Personnel 
program component lower than for any other program component. The 
on-site evaluation team members rated the level of improvement in this 
program component fifth relative to the six. program components. These two 
findings tend to agree with each other in terms of the relative level of 
improvement for the Personnel program component. f 

A significant finding in the Program Management program component 
was in the reported improvement, by LEA personrtel and second visit ETRs , 
in support by the LEA Board of Education for occupational education, 
Thi* LEA pt-rsonnt.;! and on-site ovaluation toam members agreed on the relative 
level (sornt'wha.L moderatu) of improvement for the Prpgram Management 



program ccxnponent. The LEA personnel rated, the level of improvement for • 
this program component fourth relative to the other program components; 
the team members rated the level of improvement third. 

For the program component Use of Community Resources, LEA personnel 
reported no significant improvement in the working relationship between 
occupational instructors and members of the community. This finding was 
somewhat contradicted by the analysis of the ETRs, The ETRs for the . 
second on-site evaluation indicated an improvement in the establishment 
and use of local advisory commit tees • Bowling (1973) found that piost 
evaluated LEAs (first on-site evaluation) had reported that the LEAs were 
planning to establish and utilize advisory coranittees more than they had 
in the past. An improvement was found in this study in the use of advisory 
committees. 

As with the Program Management program component, the LEA personnel 
and on'^site evaluation team members agreed to the relative level of 
improvement for the Use of Community Resources. The LEA personnel rated 
the level of improvement third relative to the other program components; 
the team members rated the level of improvement second. 

Major Findings 

Both LEA personnel and on-site evaluation team members agreed that 
there had been some degree of improvement in all occupational education 
program components and that the on-site evaluation had contributed to that 
improvement . 

Both LEA personnel and team members reported that the on-sitie evaluations 
were a beneficial and worthwhile investment of resources. 

Both LEA personnel and on-site evaluatic/n team members reported that 
part t' cipa tion in the on-site evaluation was of benefit to ^themselves and 
to their LEA occupational education program. 

There xvas a high degree of agreement between the content of the 

id 



Evaluation Team Reports written by the on-site evaluation teams at the time 
of the on-site visits and the team members expressed viewpoints at a point 
in time following the on-site visit* - . 

There was a high level of agreement between the viewpoints expressed 
*hy the LEA personnel via the PEI-F at the time of the orientation meeting 
held prior to the on-site visit and the viewpoints expressed via a^follow- 
tip (^questionnaire completed at a point in time following the on-site visit. 

LEA personnel with an administrative assignment generally rated the 
working relationships among the various LEA personnel grpups and the board 
of education and the benefits of the on-site evaluation higher than those 
persons without an administrative assignment. ^ 

Guidance personnel generally rated all aspects of the occupational 
education program and the on-site evaluation phase lower than other LEA 
personnel. 

LEA personnel became involve({ in a higher level of ii^teractions, 
meeting with advisory coamittees and other activities related to program 
Improvement while preparing for the on-site visit. Guidance personnel 
were the least involved of all LEA groups. Those LEA persons who had 
previously served on aa on-site evaluation team were more involved in the 
on-site visit of their own LEA than were those who had not served on a 
team. 

LEA personnel tended to rate the level of improvement in components 
in which they were involved higher than they rated the other program 
components where they were less involved. 

LEA personnel and on-site evaluation team members rated the level of 
Improvement in the six program components as being from somewhat more than 
lit: tie to moderate. 

LEA personnel and on-site evaluation team members generally agreed as 
to the relative level of improvement in each of the. program components. 

The relative amount of improvement by program component over the five 
year time period was, highest to lowest: Occupation Programs, Use of 
Community Resourses, Planning and Evaluation, Program Management, Student 
Services, and Personnel. 

Student membership in vocational clubs has declined from the time of 
the first on-site evaluation and relatively less instructor involvement 
was reported for this area. 

Cone lusions 

Based on the results of this study and the discussion presented, the 
following conclusions seem appropriate. 
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On-site evaluations conducted by evaluation teams contribute to^ 
improvement in LEA occupational education programs. 

Guidance personnel are less involved in the occupational education 
programs than other groups of LEA personnel and report the least benefit 
from participation in the on*site evaluation process. Guidance personnel 
are not extensively or actively involved in interactions with occupational 
education personnel or on-site evaluation teams* 

Activities initiated by DAVTE as a result of concerns brought out by 
^ the earlier on-site evaluations have been responded to and are reflected 
* in the improvement jf occupational education programs. 

Investment in the on-site evaluation process for occupational education 
.programs returns various types of benefits to the LEA occupational programs 
and personnel. 

LEA personnel who serve as an on-site evaluation team member tend to 
gain benefits for their own program and to become more involved in their 
own prQgram. ; 



Outside personnel serving on on-si;a evaluation teams and LEA 
personnel have a high level of agreement concerning the nature of the 
LEA programs and their chang^es over time. 

There has been an increased emphasis on evaluation of the individual 
occupational programs by the on-site evaluation teams as a part of the 
on-site evaluation* 

The evaluation team reports for the second on-site evaluation are far 
more extensive thah for the fipst on-site evaluation. 



The DAVTE on-site evaluation should be continued with some modifications. 

DAVTE should establish ne^hanisms and procedures to gain a more wide- 
spread involvemriic across all l£A personnel groups for participation in the 
on-site evaluation team process. 

DAVTE shouM establish mechanisms and procedures to gain more involve- 
ment of guidance p'^rsonnel both in the operating occupational education 
programs and on the on-site evaluation teams. 

DAVTE should consider recruiting and assigning additional guidance 
personnel i:o the on-site evaluation teams so that at least one guidance 
counselcr will be on each on-oite evaluation team. 

With the increased emphasis placed on the Individual occupational 
programs as a part of the on-site evaluation?, It Is recommended that DAVTE 
evaluate the present criteria used in jstablishing the on-site evaluation 






4H 



44 

teams. It is possible that having at least one team member from each of 
the five program areas on the team will Improve the on*slte evaluation 
process and the local occupational education programs* 

DAVTE should develop a program of activities directed towards ^ 
improving the current low Involvement of occupational education students 
in vocational clubs if participation in such activities is a goal of DAVTE. 

DAVTE should establish a computer based data library for the data 
collected as a part of the on-site evaluation phase to provide a basis for 
further studies of the TPS on-site evaluation. 

Further studies should be conducted to determine how LEA personnel are 
Involved In implementing the Improvements recommended by the on*-slte evaluation 
team and the development of the LEA occupational education program and the 
One and Five Year Flan after the on-site evaluation Is completed. 
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